The Dow Jones industrial average extended gains after the conclusion of Yellen's midday remarks to briefly trade more than 140 points higher, with Boeing (BA) contributing the most to gains as most constituents advanced.

The S&P 500 held above the psychologically key 2,100 level to trade around its highest of the year so far. Energy rose more than 1.5 percent to lead advancers while utilities and telecommunications were the only decliners.

"You have oil prices moving higher and that's been helpful," said Quincy Krosby, market strategist at Prudential Financial.

"The market right now needs to have growth. It also needs to have earnings revisions moving in a positive direction, which they are," she said. "If the GDP forecast continues to stay above two percent, I think it satisfies the need for the market at this valuation."

Yellen's midday remarks at the World Affairs Council of Philadelphia did not give a specific time period for the next hike but said the Fed funds rate probably needs to rise gradually over time. Yellen said while the overall labor market situation has been quite positive, Friday's report was "disappointing."

The market "realized pretty quickly she wasn't talking about a rate hike next week," said Robert Pavlik, chief market strategist at Boston Private Wealth. "I think one's coming but that shouldn't knock the market off this level. A rate hike followed by an indication that another would soon follow, that would knock the market off."

The U.S. dollar index came off session lows to trade mildly lower, with the euro near $1.137 and the yen around 107.4 yen against the greenback as of 3:15 p.m. ET. The dollar index fell nearly 1.7 percent Friday in its worst day since early December.

Treasury yields briefly came off highs before holding near earlier levels, with the 2-year yield (U.S.:US2Y) around 0.80 percent and the 10-year yield (U.S.:US10Y) around 1.72 percent.

"I still think July is still a possibility. I think June is off the table," said John Caruso, senior market strategist at RJO Futures.

On May 27, Yellen had said a rate hike in the next few months would probably be appropriate.

U.S. crude oil futures posted their best settle since July, while brent traded above $50 to hit a fresh high for the year so far. The gains in oil were supported by a soft U.S. dollar and near-term supply disruptions from attacks on Nigerian oil infrastructure.

"This (rise in oil prices) is all very positive for that sector of the economy that has been a concern from a credit risk point of view," said Marc Chaikin, CEO of Chaikin Analytics.

He also noted support for stocks overall from gains in small-cap stocks. The Russell 2000 held about two-thirds of a percent higher in midday trade.

In economic news, Reuters reported the Fed's index on labor market conditions fell in May for a seventh-straight month to minus 4.8, the lowest since May 2009.

Earlier, St. Louis Fed President James Bullard said in an interview with The Wall Street Journal that it's a "fair assessment" the chance of a June rate rise is now much lower and that it would be better for the Fed to raise rates on the back of good economic news.

Separately, Atlanta Fed President Dennis Lockhart told Bloomberg Television the U.S. central bank should wait until July before considering whether to hike interest rates, citing a weak May jobs report and potential disruptions from Britain's June 23 vote on whether to leave the European Union. He is a non-voting member of the FOMC.

Boston Fed President Eric Rosengren, a voting member, said the U.S. economy's recent recovery has moved the Fed closer to raising rates, although May's "disappointing" employment report might delay the timing of the next hike.

On Friday, U.S. stocks closed lower with financials lagging after the May jobs report disappointed with a headline figure of just 38,000, the smallest gain since 2010 and well below expectations of 162,000. The unemployment rate fell to 4.7 percent, primarily due to a decline in labor force participation. Average hourly wages rose 0.2 percent from the prior month for a year-on-year increase of 2.5 percent.

The major European indexes were slightly higher, while bank stocks lagged.

Asian stocks closed mixed, with the Shanghai composite and Nikkei 225 slightly lower and the Hang Seng about 0.4 percent higher.

In afternoon trade, the Dow Jones industrial average (Dow Jones Global Indexes: .DJI) rose 141 points, or 0.8 percent, to 17,949, with Boeing leading advancers and Home Depot (HD) the only decliner.

The S&P 500 (^GSPC) traded up 14 points, or 0.68 percent, to 2,108, with energy leading nine advancers and utilities the only decliner.

The Nasdaq (^IXIC) composite traded up 33 points, or 0.68 percent, to 4,975.

The CBOE Volatility Index (VIX) (^VIX), widely considered the best gauge of fear in the market, traded higher near 13.5.

About three stocks advanced for every decliner on the New York Stock Exchange, with an exchange volume of 505 million and a composite volume of nearly 2.3 billion in afternoon trade.

U.S. crude oil futures for July delivery settled 2.2 percent higher at $49.69 a barrel on the New York Mercantile Exchange.

Gold futures for August delivery rose $5.30 to $1,248.20 an ounce as of 12:02 p.m. ET.

—Reuters contributed to this report.