{"action":"created","issue":{"url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/95","repository_url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io","labels_url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/95/labels{/name}","comments_url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/95/comments","events_url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/95/events","html_url":"https://github.com/zhangyachen/zhangyachen.github.io/issues/95","id":188960239,"node_id":"MDU6SXNzdWUxODg5NjAyMzk=","number":95,"title":"redis hash结构 遍历某一个key下所有的(field,values)的方法","user":{"login":"zhangyachen","id":14103319,"node_id":"MDQ6VXNlcjE0MTAzMzE5","avatar_url":"https://avatars1.githubusercontent.com/u/14103319?v=4","gravatar_id":"","url":"https://api.github.com/users/zhangyachen","html_url":"https://github.com/zhangyachen","followers_url":"https://api.github.com/users/zhangyachen/followers","following_url":"https://api.github.com/users/zhangyachen/following{/other_user}","gists_url":"https://api.github.com/users/zhangyachen/gists{/gist_id}","starred_url":"https://api.github.com/users/zhangyachen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhangyachen/subscriptions","organizations_url":"https://api.github.com/users/zhangyachen/orgs","repos_url":"https://api.github.com/users/zhangyachen/repos","events_url":"https://api.github.com/users/zhangyachen/events{/privacy}","received_events_url":"https://api.github.com/users/zhangyachen/received_events","type":"User","site_admin":false},"labels":[{"id":480567235,"node_id":"MDU6TGFiZWw0ODA1NjcyMzU=","url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/labels/redis","name":"redis","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-13T09:09:17Z","updated_at":"2018-10-15T06:34:59Z","closed_at":null,"author_association":"OWNER","body":"redis的hash结构中存储了如下的数据：\r\n```php\r\n$input = array(\r\n    \"key\" => $key,      //唯一的key值\r\n    \"qid\" => $qid,      //问题id\r\n    \"value\" => $startTime_$endTime,     //开始时间_结束时间\r\n)\r\n```\r\n需求：每天凌晨跑定时脚本，跑出一个key下的所有qid，判断当前时间与value，当`$endTime<time()`时，即为过期qid，需要删除它。\r\n\r\n所以我们怎么找出所有的(field,values)呢，我想到了如下的方法：\r\n* 通过redis的`HGETALL`命令。\r\n* 在DB中存储一份key=>qid的集合。\r\n* 通过redis的`HSCAN`命令。\r\n\r\n#### 通过redis的HGETALL命令\r\n```php\r\n$input = array(\r\n    \"key\" => $key,\r\n);\r\n\r\n$ret = $objRedis->HGETALL($input);\r\nvar_dump($ret);\r\n```\r\nresult:\r\n```\r\narray(3) {\r\n  [\"ret\"]=>\r\n  array(1) {\r\n    [\"business_AdvancedPackageOne_2\"]=>\r\n    array(16) {\r\n      [0]=>\r\n      array(2) {\r\n        [\"field\"]=>\r\n        string(2) \"58\"\r\n        [\"value\"]=>\r\n        string(3) \"1_1\"\r\n      }\r\n      [1]=>\r\n      array(2) {\r\n        [\"field\"]=>\r\n        string(2) \"56\"\r\n        [\"value\"]=>\r\n        string(5) \"56_57\"\r\n      }\r\n      [2]=>\r\n      array(2) {\r\n        [\"field\"]=>\r\n        string(2) \"57\"\r\n        [\"value\"]=>\r\n        string(5) \"57_58\"\r\n      }\r\n      ...\r\n    }\r\n  }\r\n  [\"err_no\"]=>\r\n  int(0)\r\n  [\"err_msg\"]=>\r\n  string(2) \"OK\"\r\n}\r\n```\r\n\r\n但是此方法有2个缺点：\r\n* 公司的redis中间层对返回数据做了64KB大小的限制，如果返回包的大小超过64KB，就会返回错误。\r\n* HGETALL会在大数据集下表现的很低效，复杂度为O(n)，n为hash表的大小。\r\n\r\n弃用。\r\n\r\n#### 在DB中存储一份key=>qid的集合\r\n简答来说，就是在DB中也存储一份key=>qid的映射。首先先查找key对应的qid有哪些，在去redis中查找，所需要的操作只是HGET操作：\r\n```php\r\n$input = array(\r\n    \"key\" => $key,      //唯一的key值\r\n    \"qid\" => $qid,      //问题id\r\n)\r\n$ret = $objRedis->HGET($input);\r\n```\r\n\r\n此方法的缺点在于：从DB中取出qid，需要循环遍历qid集合，一次次去读redis，不像HGETALL那样一次可以全部读出来。但是循环遍历操作我认为可以通过如下两个方法去解决：\r\n* 将循环遍历操作变为批量读取：\r\n```php\r\n$input = array(\r\n    'reqs' => array(\r\n        $input1,\r\n        $input2,\r\n        ...,\r\n    ),\r\n);\r\n$rpc->$cmd($input);\r\n```\r\n* 在数据库中增加startTime和endTime两个字段，直接在数据库层判断哪些qid过期，将过期的qid取出来，直接组装成数据，调用`hdel`一次性全部删除(以下是我脚本中的一段代码)：\r\n```php\r\n/**\r\n * @desc 获取当前机构行家过期的增值包1的qid集合\r\n * @param $objBusinessDB\r\n * @param $businessId\r\n * @param $currentTime\r\n * @return array\r\n */\r\nfunction getMergeQid($objBusinessDB,$businessId,$currentTime){\r\n    //过期的qid集合\r\n    $outDateQid = array();\r\n    $outDateId = array();\r\n\r\n    //查询过期的包信息\r\n    $sql = \"select * from busines_package where businessId={$businessId} and endTime<{$currentTime} and deleted=0\";\r\n    $dbRet = $objBusinessDB->query($sql);\r\n    if($dbRet === false){\r\n        Bd_Log::warning(\"select table fail.sql[\".$sql.\"]\");\r\n        return $outDateQid;\r\n    }\r\n\r\n    foreach($dbRet as $value){\r\n        $extpack = mc_pack_pack2array($value['extpack']);\r\n        //合并qid信息\r\n        if(isset($extpack['qid']) && !empty($extpack['qid'])){\r\n            $outDateQid = array_merge($outDateQid,$extpack['qid']);\r\n        }\r\n        $outDateId[] = intval($value['id']);\r\n    }\r\n\r\n    $sql = sprintf(\"update business_package set deleted=2 where id in (%s)\",implode($outDateId));\r\n    $dbRet = $objBusinessDB->query($sql);\r\n    if($dbRet === false){\r\n        Bd_Log::warning(\"update table fail.sql[\".$sql.\"]\");\r\n    }\r\n\r\n    return $outDateQid;\r\n}\r\n```\r\n```php\r\n    //获取过期的qid集合\r\n    $outDateQid = getMergeQid($objBusinessDB,$businessId,$currentTime);\r\n    //删除redis中的qid记录\r\n    $redisRet = $objDaoRedis->HDEL($redisKey,$outDateQid);\r\n    if($redisRet['err_no'] != 0){\r\n        Bd_Log::warning(\"delete advancedPackage fail.businessId[$businessId] currentTime[$currentTime]\");\r\n    }\r\n```\r\n为了防止过期的qid在第二天重复被读取，在发现有过期的qid集合时，将对应的deleted字段设置为2。这样的话就能防止被重复读取(`where deleted=0`)，提高效率。\r\n\r\n采用。\r\n\r\n#### 通过redis的`HSCAN`命令\r\n关于HSCAN命令的介绍：[SCAN](http://redisdoc.com/key/scan.html#scan)\r\n\r\n基本思想是：通过redis自身的`HSCAN`命令，循环读取一个key下的所有qid。优点很明显：\r\n* 相比于HGETALL，都能读取出一个key下所有的(field,value)。但是由于是循环读取，占用redis服务器的资源和耗时都较少。\r\n* 不用在DB中冗余存储一份key=>qid的映射，操作也相对于简单了许多。\r\n\r\n但是悲催的是，公司的redis服务层不支持hscan命令：\r\n```php\r\narray(2) {\r\n  [\"err_no\"]=>\r\n  int(405)\r\n  [\"err_msg\"]=>\r\n  string(37) \"Unknown Method: unknown method(HSCAN)\"\r\n}\r\n```\r\n\r\n放弃。\r\n\r\n综上，采用第二种方法：在DB中存储一份key=>qid的集合。"},"comment":{"url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/comments/429724714","html_url":"https://github.com/zhangyachen/zhangyachen.github.io/issues/95#issuecomment-429724714","issue_url":"https://api.github.com/repos/zhangyachen/zhangyachen.github.io/issues/95","id":429724714,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTcyNDcxNA==","user":{"login":"vindurriel","id":1803110,"node_id":"MDQ6VXNlcjE4MDMxMTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1803110?v=4","gravatar_id":"","url":"https://api.github.com/users/vindurriel","html_url":"https://github.com/vindurriel","followers_url":"https://api.github.com/users/vindurriel/followers","following_url":"https://api.github.com/users/vindurriel/following{/other_user}","gists_url":"https://api.github.com/users/vindurriel/gists{/gist_id}","starred_url":"https://api.github.com/users/vindurriel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vindurriel/subscriptions","organizations_url":"https://api.github.com/users/vindurriel/orgs","repos_url":"https://api.github.com/users/vindurriel/repos","events_url":"https://api.github.com/users/vindurriel/events{/privacy}","received_events_url":"https://api.github.com/users/vindurriel/received_events","type":"User","site_admin":false},"created_at":"2018-10-15T06:34:59Z","updated_at":"2018-10-15T06:34:59Z","author_association":"NONE","body":"你百度的吧？64KB : )"}}
{"actor":{"display_login":"vindurriel"}}