python - unpack_from requires a buffer of at least 4 bytes -

i receiving packet client, consisting of many fields. read fields successfully, when comes last field tag_end, python gives me error:

unpack_from requires buffer of @ least 4 bytes not found.

this code:

def set_bin(self, buf):     """reads vector of bytes (probably received network or     read file) , tries construct packet structure     it, reading each packet member buffer.      somehow deserializing packet.     """     assert isinstance(buf, bytearray), 'buffer type not valid'     offset = 0      print("$$$$$$$$$$$$$$$$ set bin $$$$$$$$$$$$$$$$$")      try:         (self._tag_start, self._version, self._checksum, self._connection_id,          self._packet_seq) = packet.packer_1.unpack_from(str(buf), offset)     except struct.error e:         print(e)         raise deserializeerror(e)     except valueerror e:         print(e)         raise deserializeerror(e)           #i=4 h=2 b=1     offset = packet.offset_guid     #14 correct     self._guid = buf[offset:offset+packet.uuid_size] #14-16 correct      offset = packet.offset_guid + packet.uuid_size      print("$$$$$$$$$$$$$$$$ guid read  $$$$$$$$$$$$$$$$$")      try:        (self._timestamp_sec, self._timestamp_microsec, self._command,         self._command_seq, self._subcommand, self._data_seq,         self._data_length) = packet.packer_3.unpack_from(str(buf), offset)     except struct.error e:         print(e)         raise deserializeerror(e)     except valueerror e:         print(e)         raise deserializeerror(e)      print("$$$$$$$$$$$$$$$$ timestamps read $$$$$$$$$$$$$$$$$")    offset = packet.offset_authenticate     self._username = buf[offset:offset + self.username_size]        #saman     offset += self.username_size      print("$$$$$$$$$$$$$$$$ username read  $$$$$$$$$$$$$$$$$")      self._password = buf[offset:offset+self.username_size]     offset += self.password_size      print("$$$$$$$$$$$$$$$$ password read $$$$$$$$$$$$$$$$$")      self._data = buf[offset:offset+self._data_length]     offset = offset + self._data_length      print("$$$$$$$$$$$$$$$$ data read  $$$$$$$$$$$$$$$$$")      try:         (self._tag_end,) = packet.packer_4.unpack_from(str(buf), offset)     except struct.error e:         print(e)         raise deserializeerror(e)     except valueerror e:         print(e)         raise deserializeerror(e)      print("$$$$$$$$$$$$$$$$ tag end read $$$$$$$$$$$$$$$$$")       if len(buf) != packet.packer.size + self._data_length:         print('failed deserialize binary data correctly , construct packet due data')     else:         print('@@@@@@@@@@@@@@@ deserialized successfully') 

and constants used in code:

struct_format_str   = r'=ihihh 16b iihhhhh 6c 9c' #saman struct_format_str_1 = r'=ihihh' struct_format_str_2 = r'=16b' struct_format_str_3 = r'=iihhhhh' struct_format_str_4 = r'=i' struct_format_str_5 = r'=6c' struct_format_str_6 = r'=9c' uuid_size   = 16 offset_guid = 14 #offset_data = 48     #shifting offset data 15 char       offset_authenticate = 48 packer   = struct.struct(str(struct_format_str))     #saman packer_1 = struct.struct(str(struct_format_str_1)) packer_2 = struct.struct(str(struct_format_str_2)) packer_3 = struct.struct(str(struct_format_str_3)) packer_4 = struct.struct(str(struct_format_str_4)) packer_5 = struct.struct(str(struct_format_str_5)) packer_6 = struct.struct(str(struct_format_str_6)) bytes_tag_start = packer_4.pack(tag_start) bytes_tag_end   = packer_4.pack(tag_end) 

and initialization of packet object, initializes fields:

def init(self, **kwargs): if 'buf' in kwargs: self.set_bin(kwargs['buf']) else: assert kwargs['command'] in packet.rtcinet_commands.values() , kwargs['subcommand'] in packet.rtcinet_commands.values(), 'undefined protocol command' assert isinstance(kwargs['data'], bytearray), 'invalid type data field' field in ('command', 'subcommand', 'data'): setattr(self, '_' + field, kwargs[field])

    self._tag_start = packet.tag_start     self._version = packet.version_current % (packet.ushrt_max + 1)     self._checksum = packet.crc_init     self._connection_id = kwargs.get('connection_id', 0) % (packet.ushrt_max + 1)     self._packet_seq = packet.packet_seq     packet.packet_seq = (packet.packet_seq + 1) % (packet.ushrt_max + 1)     self._guid = uuid.uuid4().bytes     dt = datetime.datetime.now()     self._timestamp_sec = int(time.mktime(dt.timetuple()))     self._timestamp_microsec = dt.microsecond     # self._command = kwargs['command']     self._command_seq = kwargs.get('command_seq', 0)     # self._subcommand = kwargs['subcommand']     self._data_seq = kwargs.get('data_seq', 0)     self._data_length = len(kwargs['data'])      self._username = packet.username            #saman     self._password = packet.password 

i have made sure read fields in right order, written in packet client program. still couldn't manage solve problem.

do have idea how solved?

the problem seems you're converting things str on place no reason.

in places, packer_1 = struct.struct(str(struct_format_str_1)), makes code less readable , understandable, doesn't affect actual output. example, struct_format_str_1 str, str(struct_format_str_1) same str.

but in other places, it's far worse that. in particular, @ lines packet.packer_1.unpack_from(str(buf), offset). there, buf bytearray. (it has be, because assert it.) calling str on bytearray gives string representation of bytearray. example:

>>> b = bytearray(b'abc') >>> len(b) 3 >>> s = str(b) >>> s "bytearray(b'abc')" >>> len(s) 17 

that string representation not going have same length actual buffer you're representing. it's no wonder errors length being wrong. (and if got unlucky , didn't have such errors, you'd reading garbage values instead.)

so, should convert bytearray struct module can handle? nothing! the docs say:

several struct functions (and methods of struct) take buffer argument. refers objects implement buffer protocol , provide either readable or read-writable buffer. common types used purpose bytes , bytearray…