Thursday, January 24, 2019

Storing Items With Over 100K Characters in Cache


Google Apps Script limits the number of characters stored to 102,400. If you try to store a value that exceeds this limit you will get the following error:

Argument too large: value (line ##, file scriptfilename)

This script overcomes that limit by breaking the string into chunks of individual strings that are each under the limit. If the string is already under the limit the string will be stored under the assigned key name. If the string goes over the limit the key name is appended by an index number starting at zero. For example, if a string has 250,000 characters and the key name is table, the string will be parsed and stored in cache in three strings with key names of table_0, table_1, and table_2. An index reference is also saved with the extension _idx so the program will know how many keys to retrieve. This function may be used on strings or JSON objects. The function getCacheChunks(key) is used to retrieve keys stored in cache using this method. Google Apps Script also limits the file size of files stored in cache to 100K. If this barrier is exceeded, try using a limit smaller than 100,000 characters. The following script stores values in user cache.  If you want to store on the document or script level change CacheService.getUserCache() to CacheService.getDocumentCache()  or CacheService.getScriptCache() , respectively. See Google's CacheServices documentation for more information.

putCacheChunks


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
function putCacheChunks(str,key,lim,exp) {
  var cache = CacheService.getUserCache();
  if(lim === undefined){lim = 100000} 
  if(exp === undefined){exp = ''} else{exp = ',' + exp}
  var len = str.length  
  var num = Math.floor(len/lim)
  var chunk = ''
  if(num == 0){cache.put(key,str)}else
    for(var a = 0; a<=num; a++){
      if(a == 0){cache.put(key + '_idx',num + exp)}
      chunk = str.slice(lim * a,lim * (a+1)); 
      cache.put(key + "_" + a , chunk +exp)}
  }

Arguments:
  • str (string): the value to be stored.  If storing an object use JSON.stringify() before passing the argument to the function.
  • key (string): the key name assigned to the string to be stored in cache
  • lim (integer): optional --Limit on the number of characters in each value stored.  The default is 100,000 (this can be changed in the script on line 4).  If the 100k file size limit is hit, try using a smaller limit.
  • exp (integer): optional -- The expiration in seconds.  The length of time before an item in cache expires.  The default  is 600 seconds (10 minutes).  
Return Values
  • None

Retrieving Items Stored In Cache Using putCacheChunks()

If values were stored in cache using putCacheChunks() then use this function to retrieve them.  If the value was stored as one single key, the value will be returned as originally stored. If the value was stored under multiple keys because the number of characters exceeded cache storage limits, the function will retrieve the values assigned to each key as individual strings and glue them together.  The combined string will be returned. If the resulting string is a JSON object, the object will be returned.

getCacheChunks


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
function getCacheChunks(key){
var cache  = CacheService.getUserCache()
var solo = cache.get(key)
var str = ''
if(solo){
  str = solo
}else{var num = cache.get(key + '_idx');
      if(!num){return null;}
      for(var a = 0; a<=num; a++){
        str += cache.get(key + '_' + a)
      }}
  try
  {var obj = JSON.parse(str); return obj;}
  catch(err)
  {return str;}
}

Arguments
  • key (string): the key name assigned to the string to be stored in cache
Return Values
  • If the key name could not be found, or if the key expired, null is returned.
  • If the string can be converted to an object using JSON, an object will be returned.
  • Otherwise the string will be returned in-tact as one value.  


on
Labels: