You can request an American Airlines seat upgrade with Avios either at the time of booking or later on, depending on if there's availability. There must be an award seat available on the flight for the upgrade to clear.

Yes, you can use British Airways Avios to upgrade on American Airlines, but the rules are quite specific. To be eligible, the flight must be marketed and operated by American Airlines, and your original ticket must be a "published fare" in a qualifying class (usually full-fare Economy or Premium Economy). You cannot use Avios to upgrade from "Basic Economy" or from tickets booked using miles. To process the upgrade, you must do so through the British Airways website or call center, and there must be "Award Space" (specifically 'A' or 'C' class) available in the next higher cabin. The "cost" in Avios is usually the difference between the Avios price of the cabin you booked and the cabin you want to upgrade to. Because this relies on specific partner availability, it can be harder to find than a standard BA upgrade, so it is best to check for availability before you buy your initial ticket if the upgrade is your primary goal.