lightcache / src / todo

The default branch has multiple heads

v0.1
--------
[+] in need of some protocol message to set some core settings dynamically. This is very useful for testing purposes.
    CHG_SETTING <setting_name> <setting_value>
[+] jemalloc is a dependency from now on. maybe disable it by a flag later.
[*] response structure which should be a linked list and a header in it.
[+] STATS command.
[+] TEST idle_client working with SET_SETTING in tests.
[+] YES IT IS ABLE TO.check if valgrind is able to detect offbyone errors buffer overruns? if not
	check for buffer overrruns. 
[+] exclude log.c.
[+] some kind of freelist for basic structures. (exponantially growing)
[+] move the headers to lightcache.h
[+] network byte ordering for response/request objects and python tests also should include endianness.
	python struct module directly writes integers without respect to the endianness. use htonl() before
	sending and nhtonl() after receiving data for integers. Strings, obviously are same. 8-bit integers
	are agreed to be the smallest byte size on C, so endianness do not affect these, obviously.
[+] prepare_response should accept a param with allocate_bytes for allocating the sdata array.
[+] AFAIU, memcache uses strtoull() and ntohll() for 64 bit and strtoul() and ntohl() for 32 bit integers.
	so, just decide which to use.
[+] send invalid/boundary integers for GET, CHG_SETTING, GET(...) functions shall return None on error.
	server generally disconnects in invalid situations.
[+] kqueue integration.
[+] use predefined macros to compile lightcache. Use only makefile like redis.
[+] fprintf(xxx)'s should end with \r\n.
[+] kqueue pass TESTS. Getting broken pipe errors on some occasions.
v0.2
---------
[*] TOO HARD from now on and memcached is also not doing like this.Subsequent read/write should be possible at the same time. And i don't expect any valid use case for this requirement. If you see one write here.
[*] use HHASH algorithm same as memcached. Bob Jenkins is the expert there:)
    Maybe improve in next versions, currently it is ok using one_at_a_time hash.
[+] use DEBUG macro for debugging the flow. Currently they are fprintf(stderr,)
[+] req per sec/sent per sec stats, and total sent/received stats.
[*] GET_MANY after all of above is working. It seems redundant as client can implement this functionality with n pipelined request with a single
    connection.
[+] memcache accepts a unix domain socket path as a param and does REUSEADDR, KEEPALIVE, LINGER on both socket types. Also, TCP_NODELAY is set for
    normal berkeley sockets.
[+] implement unix domain sockets.
[+] Gprof is only working when program exists normally, without SIGABRT or SIGKILL. SO find a way to gracefully shutdown a
    lightcache server. aNy maintenance window?
[+] responses should have an interface defining return codes from the server such as KEY_NOTEXISTS or PARAM_INVALID
    Current test cases have assertDisconnect on protocol errors and this should not be the case. disconnect on server 
    side should only happen on server side errors such as not being able to alloc mem or so...
[+] TEST: Invalid setting CHG_setting
[+] Tests need refactoring. maybe a testprotocol file definind protocol messages errcodes...etc. Combine asertErrcode somehow.
[+] manual testing interface. 
[+] write uptime and version info in stats.
[+] assert in test suite shall show params when failed.
[+] test_send_overflow_data is failing. tHe problem is taht it recursively sends INVALID_PARAM_SIZE again and again. Becase the data len
    is so small. the remaning data is treated like a different message. Change the test.
[*] problem when run autotest and manual test together at the same time, when auto tests finish, manual gives unpack error.
    think it is because idle timeout is set to 2 in auto and inifinite in manual tests when auto run after manual it disconnects the manual
    client.
[+] send_nbytes with 1 byte fails on autotests somehow. Dont understand why?    
    Boils down to this: Somehow send_nbytes cannot be finished. This may be related 
    with the blocking unix socket problem.
    CMD_CHG_SETTING request
    Invalid setting received :invalid_setting_key
    sending err response:1
    send_nbytes called.[left:8]
    write completed
    send_nbytes called.[left:7]
    write completed
    read_nbytes called.    
    It seems like that when a new connection comes in previous send events are cleared.    
    Problem seeems to be in client side, when we call recv(NBYTES) it does not wait for NBYTES this behavior is the problem
    when we send 1 bytes on server side.
[+] check non-blocking rc and debug.
[+] latest freelist from yappi included.
[*] verify socket_path is valid. BIND gives error.
[+] show errcode strings defniitons in manual tests.
[+] when using unix sockets send_overflow_data kind of tests put write() into blocking mode. Is socket remain blocking?
    unix socket write blocking problem? in my tests I received Connection reset by peer after test_overflow_header
    and no conn is accpted from then on.  
    
    boils down to this: (debugged with strace)
    16:06:12.295028 time(NULL)              = 1311685572
    16:06:12.295072 epoll_wait(4, {{EPOLLOUT|EPOLLHUP, {u32=151467988, u64=151467988}}, {EPOLLIN, {u32=151468084, u64=151468084}}}, 10, 1000) = 2
    16:06:12.295122 time(NULL)              = 1311685572
    16:06:12.295168 write(1, "send_nbytes called.[left:8, fd:7"..., 35send_nbytes called.[left:8, fd:7]
    ) = 35
    16:06:12.295224 write(7, "A", 1)        = -1 EPIPE (Broken pipe)
    16:06:12.295293 --- SIGPIPE (Broken pipe) @ 0 (0) ---
    16:06:12.306404 +++ killed by SIGPIPE +++
    
    strace command used to start lightcache is:
    strace -Ff -tt /home/sumerc/projects/lightcache/src/lightcache -s /home/sumerc/Desktop/deneme
    
    ** memcache ignores SIGPIPE on both deamon and tty modes.
[+] we need some kind of variable on the sh files if we are running on domain sockets or not.    
[+] maybe an strace .sh file.   
[+] all kqueue tests passed on BSD. 
[*] NOT NECESSARY.any severity number for logs that can be passed in compile time? for example we only want to trace protocol messages or do a 
    deeper trace.
[+] gdb.sh now accepts cmd line params.
[+] refactor/optimize kqueue. Before refactoring write a test for epoll that sends subsequent packets before receiving data
	to see what happens and use that info to refactor kqueue. IMHO, if event_set() is called with EVENT_READ then this should
    be the only	event registered in the pollfd, others shall be discarded.Verify that assumption with epoll and integrate same
	logic with kqueue. 
[*] run and inspect debugmem.sh for mem leaks or buf. overflows. Not very successfull, instead using my own memleak testing.
[+] a shutdown command for the server to gracefully shutdown itself.(Only available in DEBUG version) 
    not necessary as i am able to detect memleaks.
[+] delete command.
[+] flush_all command. memcache uses an optional timeout to flush the data but IMHO in LC we dont need that in current versions. 
[+] TEST: flush all. add more related tests. Maybe to clear all keys on some tests, may be needed dunno.
[+] add more stats mimic memcached.
    Name              Type     Meaning
    ----------------------------------
    pid               32u      Process id of this server process
    uptime            32u      Number of seconds this server has been running
    time              32u      current UNIX time according to the server
    version           string   Version string of this server
    pointer_size      32       Default size of pointers on the host OS
                               (generally 32 or 64)
    rusage_user       32u:32u  Accumulated user time for this process 
                               (seconds:microseconds)
    rusage_system     32u:32u  Accumulated system time for this process 
                               (seconds:microseconds)
    curr_items        32u      Current number of items stored by the server
    total_items       32u      Total number of items stored by this server 
                               ever since it started
    bytes             64u      Current number of bytes used by this server 
                               to store items
    curr_connections  32u      Number of open connections
    total_connections 32u      Total number of connections opened since 
                               the server started running
    connection_structures 32u  Number of connection structures allocated 
                               by the server
    cmd_get           64u      Cumulative number of retrieval requests
    cmd_set           64u      Cumulative number of storage requests
    get_hits          64u      Number of keys that have been requested and 
                               found present
    get_misses        64u      Number of items that have been requested 
                               and not found
    evictions         64u      Number of valid items removed from cache                                                                           
                               to free memory for new items                                                                                       
    bytes_read        64u      Total number of bytes read by this server 
                               from network
    bytes_written     64u      Total number of bytes sent by this server to 
                               network
    limit_maxbytes    32u      Number of bytes this server is allowed to
                               use for storage. 
    threads           32u      Number of worker threads requested.
                               (see doc/threads.txt)
[*] hget() can be optimized more as strcmp() can be optimized. 
    GCC is already optimized.
[+] send success codes for every msg???
[*] in the current slab allocator pointer artihmteic is used on void * 's .
    May that cause problem. Inspect that. No it doesnt cause. It is the best way
    pointer artihmetic is to restrict the arithmetic size. Normally it is one byte
    we cast to char * to indicate that which is correct of course for our case.
[*] Implement GETQ/SETQ.
[*] TEST: GETQ/SETQ
[+] BUG: AUTOTESTS fail with new integration. Freelist removal causes double free 
    on some cases.
[+] BUG: LC cannot run slab alloc failure with 1 MB of memory.
[+] slab_alloc() for dynamic allocations and free list should be implemented according to
    wikipedia: http://en.wikipedia.org/wiki/Free_list.
[*] MAKE: Makefile shall contain make tests which will run autotests.sh with TCP and domain sockets.
    Unit tests for C integrated as running under the shell scripts.
[+] MAKE: LITTLE_ENDIAN flag should somehow be passed to Makefile. With configure script maybe?
[+] TEST:Valgrind is necessary, especially after we removed the freelist.
[+] maybe prepare_response can be merged into send_response() function for better
    design.
[+] Valgrind uninited var warnings. 
[*] Slab allocation solves the fragmentation problem. Redis makes a contest to stress the mem.
    fragmentation. Can we have such chance to observe fragmentation via some tests? 
    Not easy as redis because we support slab re-assignment that makes things harder.
    Just observe the fragmentation manually.
[+] TEST: more delete related tests especially in memleaks.
[-] show the rlimit hardlimit of the system like we do in esserver.
[-] any way to exclude math.h that requires -lm dependency for the linker?
[-] lightcache clients shall be moved to clients folder. Maybe later make another rep.
    for all of them. tests will have dependencies on them.
[-] TEST/INSPECT: dynamic alloc. failures.(OUT_OF_MEMORY)

[-] write a php client, there is an unpack function for PHP4-5.
[-] ruby client. there is an unpack() cmd.
[-] solve distribution problem on client side. http://amix.dk/blog/post/19370.
    there is also a video for DHTs on google tech talks but cannot remember its name. 
[-] what does this mean: socket Numerical result out of range (read_error) in var/log/syslog.
[-] TEST: add tests for uncompleted messages where we have key but data is missing etc. Write test for most of them,, if possible all of them.
[-] TEST:more memleak tests required but logic is some kind of hard because all items alloc'd in LC are
    recyclable. Think more carefully about that tests.
[-] RE-TEST again with LINGER OPtion as memcached.test_protocol suite failing when run in remote. send_overflow_data is not returning from assertDisconnect() somehow when lightcache is run remotely. Remote system is CentOS 64 bit.
[-] Program received signal SIGFPE, Arithmetic exception. disable this. 
	if ( (stats.mem_used * 100 / settings.mem_avail) > LIGHTCACHE_GARBAGE_COLLECT_RATIO_THRESHOLD)	
[-] more tests & more tests.
[-] We may need tests for MacOSx
[-] any optimization window for malloc()'s?
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.