The lawsuit accuses Google and its popular online video unit of "massive intentional copyright infringement" for allowing users to upload popular shows, threatening ambitions to make YouTube a major entertainment and advertising outlet.
The legal challenge from Viacom, home to the MTV and Comedy Central channels, also suggested a wider battle between traditional and Internet media companies that now compete for audiences and advertising dollars.
"This is a seminal event in Media-Internet relations ... and how the value of content will be clarified in the online medium," wrote UBS analyst Aryeh Bourkoff in a client note.
Shares in Viacom slipped 9 cents to close at $39.48 on the New York Stock Exchange and Google shares fell $11.72, or 2.6 percent, to $443.03 on Nasdaq.
Viacom has been the most vocal critic of YouTube during months of negotiating over payment for use of its programming. The Sumner Redstone-controlled company last month demanded YouTube pull over 100,000 video clips uploaded by users
"YouTube's strategy has been to avoid taking proactive steps to curtail the infringement on its site, thus generating significant traffic and revenues for itself while shifting the entire burden -- and high cost -- of monitoring YouTube on to the victims of its infringement," Viacom said.
YouTube does not prevent copyrighted content from being uploaded onto its site, but will take material down at the request of copyright owners.
Google said it was confident that YouTube respects the copyrights at issue in the Viacom case.
"We will certainly not let this suit become a distraction to the continuing growth and strong performance of YouTube," Google said in a statement.
General Electric Co.'s (GE.N: Quote, Profile, Research) majority-owned NBC Universal and News Corp. (NWSa.N: Quote, Profile, Research) have also criticized YouTube's policies on copyright protection but stopped short of legal action, testimony to the dilemma of media companies forced to choose between embracing a fast-growing outlet for younger audiences and trying to build competing Web vehicles themselves.
"We've dealt with YouTube on a case by case basis to have content taken down," a News Corp. (NWSa.N: Quote, Profile, Research) spokesman said, adding that the company supported Viacom's right "to protect its own content in whatever way it needs to."
Viacom found another ally in Time Warner Inc. (TWX.N: Quote, Profile, Research)
"It is clear from this lawsuit that it is time for YouTube to remove unauthorized material from its site," a Time Warner spokesman said. "We are in talks and hopeful we can work together toward a solution that would effectively identify and filter out unauthorized material and license copyrighted works for an appropriate revenue share."
Viacom contends that almost 160,000 unauthorized clips -- from excerpts of comedy talk show "The Daily Show with Jon Stewart" to pieces of children's programs like "SpongeBob SquarePants" -- have been uploaded on to YouTube's site and viewed more than 1.5 billion times.
The decision to sue Google followed "a great deal of unproductive negotiation," the company said.
Viacom filed the suit in the U.S. District Court for the Southern District of New York, seeking an injunction against further violations and damages.
Google bought YouTube last November for $1.65 billion, aiming to capitalize on its explosive audience growth built from sharing both homemade and professionally produced videos.
YouTube has reached licensing deals with major record labels, but still faces the ire of major media companies. Google has promised new technology to help identify pirated videos, but has not given a timetable for its introduction.
Any progress Viacom makes in its lawsuit could spur other companies to consider legal action against YouTube and raise new questions about the laws governing digital distribution.
"If there's anything central to Google's business model, it is being at the center of everything," said Forrester Research analyst James McQuivey. "This has the potential to put them on the periphery."
Viacom and peers like NBC Universal, in which France's Vivendi (VIV.PA: Quote, Profile, Research) owns a minority interest, are also investing heavily in their own Internet video sites to benefit from the migration of television audiences to the Web.
"There is certainly an opportunity for YouTube to do a deal with Viacom, but Viacom does not have to have a YouTube deal," said analyst Richard Greenfield of Pali Capital.
Google's dominance in Web search has made it a magnet for lawsuits by copyright and trademark holders.
The Silicon Valley company faces outstanding lawsuits in the United States and Europe by major book, magazine and online news publishers as well as small-time Web site operators.
Google has prevailed in high-profile suits against it by auto insurer GEICO -- owned by billionaire investor Warren Buffett's holding company Berkshire Hathaway Inc. (BRKb.N: Quote, Profile, Research) -- over trademark infringement, and in a demand by the U.S. Justice Department for consumer Web search data.