I assume you're talking about a mobile device of some kind which displays or transmits a one-time password derived from a shared secret and either the time or a shared counter. Such devices fall into three categories:
- A standalone device with a display.
- A standalone device with a USB (or other) interface which must be accessed via a computer.
- A general-purpose device such as a mobile phone.
The relative security of the various device types depends on your threat model.
The most obvious threat is that an attacker steals or borrows the device. In this scenario, it doesn't matter how hard it is to extract the key from the device: the attacker merely needs to be able to use it in the usual manner.
The attacker only needs to extract the key if he wants to be able to authenticate later, but wants his access potential to go undetected. With this goal, the attacker must gain temporary access to the device, extract the key, then return the device to its owner. After this, the attacker can go undetected as long as he does not make too frequent connections to the account if the OTP is derived from a counter, or indefinitely if the OTP is derived from the time.
The security of the key against an attacker with physical access only matters if you want to protect against hidden access. A standalone device may have better protection against physical attacks than a mobile phone, however such protection costs money, and authentication tokens are usually a low-cost item. High-end mobile phones do have some limited protection against physical attacks; for example, it is not so easy to extract data from an iPhone. Most mobile phones furthermore contain a SIM card, which is tamper resistant; some OTP providers can install the OTP key on the SIM. Some mobile phones contain an embedded secure element, which is similar to the SIM but controlled by the device manufacturer instead of the network operator.
Most OTP token devices display the OTP at the press of a button. A mobile phone has the opportunity to require authentication (PIN, pass-gesture, …). How much this improves security depends on whether the phone is stolen unlocked and how strong the authentication is.
There is a threat where mobile phone gets a slight edge: that the user would lose the token. This is in addition to the usability benefit of having a single device. A user who has many OTP tokens is more likely to lose one of them, or to not notice its theft for a while, than he is likely to miss his mobile phone.
A different threat is that of malware on the device. Standalone devices are pretty much immune, assuming they were clean when the user got them. Mobile phones are vulnerable; an attacker who can install malware that retrieves OTP tokens gains remote access to the one-time passwords. Devices without their own display are similarly vulnerable to malware on the computer they are plugged into, though this malware will only be able to obtain an OTP while the device is connected.