There function in libsecp256k1 that allows you to directly set the field element to 32 bytes of your choice. Does secp256k1 always return a valid group element for any 32 byte value?
secp256k1_fe_set_b32(&xpoint, bytes)
secp256k1_ge_set_xo_var(ge, &xpoint, 0)