Investors have been bullish on Rio in the past two years, pushing its share price from A$48 per share in 2016 to $82/share this week.
RBC said the major's valuation would be hit by higher costs and weaker iron prices.
"At 8.2x EV/EBITDA on our revised 2019E forecasts and with a sharply weaker near-term iron ore outlook, we expect Rio Tinto will be unable to hold its current 64% premium to sector valuations," it said.
"Cost inflation and growing potential for M&A are added challenges which will complicate the investment case for this recent perennial winner."
RBC said Jean-Sebastien Jacques could improve the company's fortunes by buying new projects.
"With an effective net cash balance sheet (following the next set of asset sale inflows) Rio Tinto has significant scope to be able to meaningfully change its future via M&A," it said.
"It is also likely to be able to maintain a high level of cash returns through this more volatile period."
RBC forecasts 2018 EBITDA at US$15.1 billion, a significant drop on last year's $18.58 billion, which the consensus forecast expects to be repeated this year.
RBC previously had Rio at ‘market perform' with a target of A$76/share, and with the downgrade it has dropped the target to $65/share, a level last seen 12 months ago.
Rio shares closed at $82.10 yesterday.