Q:

What is HEX STRING representation of BITS datatype?

How to convert BITS datatype to HEX STRING datatype? (like 0x80) What is HEX STRING representation of (0), (1), (2) BITS respectively?  The DEFVAL is '00'h. 
Composed Type: Bits
Base Type: BITS

Value List: 
large (0)
mid-size (1)
compact (2)

A:

You've probably found the correct syntax for hex and Bits data on the man page of "snmpset":
http://www.net-snmp.org/docs/man/snmpset.html
So, setting the Bits value large (0) is done like with the syntax
snmpset <flags,host etc> <oid> b 0

And to set multiple flags, like both mid-size and compact, use 
snmpset <flags,host etc> <oid> b 1,2

The latter syntax is not well-documented, I think. I found it in the mailing list archives for net-snmp.
Like I wrote in the comments, there is no "HEX STRING" datatype in SNMP. If you want to transmit binary data, such as an OCTET STRING value, then you will need to represent it somehow on the command line. Net-snmp provides an input format called "HEX STRING" which will accept hexadecimal input using the following syntax:
snmpset <flags,host etc> <oid> x 60

Incidentally, "60" in hexadecimal is identical to how SNMP would encode "bits 1,2" (the second and third bits, MSB first).
A description of how SNMP encodes and decodes BITS values is found in RFC1906 section 8. Having that knowledge, you just need to learn how hexadecimal conversion works, which is readily googleable.
Edit: I guess this is what you're looking for:
0 -> 0x80
1 -> 0x40
2 -> 0x20

This is a bit backwards compared to normal (LSB) byte encoding, but again read RFC1906 for a description of how it works.
It should be noted that if you try to set an OCTET STRING value on a BITS variable, you might expect the SNMP agent return a type error. However, since BITS is just an SMI macro based on the OCTET STRING type, they are actually encoded the same way. So, the agent should happily set the value.

