Add support for EOS 70D 1.1.1 (both revisions)

#620 Open
Repository
nikfreak
Branch
70D-merge
Repository
hudson
Branch
unified

Bitbucket cannot automatically merge this request.

The commits that make up this pull request have been removed.

Bitbucket cannot automatically merge this request due to conflicts.

Review the conflicts on the Overview tab. You can then either decline the request or merge it manually on your local system using the following commands:

hg update 7a3b5fa3f4c6
hg pull -r 70D-merge https://bitbucket.org/nikfreak/magic-lantern
# Note: This will create a new head!
hg merge 70D-merge
hg commit -m 'Merged in nikfreak/magic-lantern/70D-merge (pull request #620)'
Author
  1. nikfreak
Reviewers
Description

I separated both known 70D firmware revisions.

70D_111A is known as the "new revision" and 70D_111B is called "old revision" on the forums.

Pull request modifies several parts of the makefile system like seen on 5D3-113-and-123 branch (PLATFORM_MAP_CONFUSES_THE_MAKEFILES) But this doesn't really clean up the according platform dirs when running "make clean" in root dir

Comments (35)

  1. Audionut

    Are there hardware differences between the revisions? Could you force a specific firmware revision instead?

    1. nikfreak author

      Looks like there is some more work needed regarding makefiles. A simple make clean won't clean up the new platform paths for 70D. @a1ex could you provide an installer fir for 70D.111B? That would be helpful for trying to maintain both revisions on the forums. I have no problem in trying to solve bugs first and update this PR or recreate a new one once some progress is made.

    2. nikfreak author

      Don't think we'll see a fw update by Canon so we have to deal with two fw revisions for now. I would say let's merge it.

      • dualiso in video mode doesn't work (like 5D2 / 7D / 50D)
      • fps override doesn't work (fallback to apex)
      • raw zebras in QR doesn't work (fallback)
  2. nikfreak author

    I am always having a "block" of weird graphics in the bottom of QR. The "block" doesn't seem to change in size/height. This is how QR looks with RAW (+JPG L/M/S..) settings (screenshot was taken while trying out #define RAW_ZEBRA_TEST -> happens also w/o that):

    qr_broken.jpg

    MRAW/SRAW (+JPG L/M/S) won't cause this issue. Also FRSP doesn'thave that problem. I could:

    Would need help with the latter one

    1. Alex

      MRAW/SRAW are not causing the issue because they use YUV zebras, not RAW.

      If FRSP shows correct RAW zebras, the issue is probably the CR2/JPEG compression (or whatever in the file saving process) is reusing the same RAM area. To understand what's going on, you could get a dm-spy log, with EDMAC stubs enabled in dm-spy-extra (as on 5D2, 550D, 500D, 5D3).

      1. nikfreak author

        @a1ex yes FRSP shows them correct. Here is the dm.log as instructed. I guess interesting starting at line 4000. Took the log by using DbgMsgLog in ML-Menu.

        70D dm.log raw zebras weird in QR only

        I used these in dm-spy-extra.c:

        #ifdef CONFIG_70D  /* 1.1.1A */
            { 0x3D8BC, "StateTransition", 4 , state_transition_log },
            { 0x3DD24, "TryPostEvent", 5 },
            { 0x3D644, "TryPostStageEvent", 5 },
        
            { 0x37FF4, "ConnectReadEDmac", 2 },
            { 0x37F30, "ConnectWriteEDmac", 2 },
            //~ { 0x38540, "RegisterEDmacCompleteCBR", 3 }, 
            //~ ABOVE RegisterEDmacCompleteCBR produces black screen when taking an image but image  
            //~ is not being written and cam doesn't crash but led stays active infinitely. 
            //~ Menus are accessible but battery needs to be taken out -> canon Screen "Recording image" 
            //~ when powering off
            { 0x37DD0, "SetEDmac", 4 },
            { 0x3817C, "StartEDmac", 2 },   
        #endif
        
        1. Alex

          First thing I notice is activity in ShootBlack task. Do you have any option that subtracts a dark frame or does some sort of noise reduction?

          From the log:

          • the EDMAC used for raw capture is #0, configured to write to 0x5ca02700.
          • CR2 size is 5568x3708, so the EDMAC writes until about 0x5ec776c0 (most likely a bit more)
          • zebra corruption is from line 136 to line 174, out of 240, which means from about 0x5dd00000 to about 0x5e200000 (size about 5 MB). Numbers extremely approximate.
          • this smells like a possible suspect: CtrlSrv:ff40a0fc:18:03: [ImgPlyer] SetImageWorkMemory 1:5de10000, 2:5e810000

          I think the memory used for capturing the CR2 is freed, so the QuickReview code allocates a block from the same area.

          Do you have better luck with zebras after a burst sequence?

            1. nikfreak author

              Did some research on chdk forums and found some discussion where they encountered a "weird" raw buffer problem (not identical but the most similar I could find) http://chdk.setepontos.com/index.php?topic=11164.140

              The fix for them was (probably) to change the capturing sequence although I am unsure about that: http://chdk.setepontos.com/index.php?topic=11164.msg119759#msg119759

              Commit: https://www.assembla.com/code/chdk/subversion/commit/3978

              Back to my zebra in QR problem: As this only happens in normal shooting modes (normal being defined as all modes except FRSP and burst sequence of 8+ pics) I guess a possible fix could be to find a suitable way and modify the capturing sequence, too

          1. Alex

            I guess we should find a way to, either:

            • prevent ImgPlyer from allocating memory at 0x5de10000
            • prevent the raw buffer from being allocated at 0x5ca02700

            For 1, the address is hardcoded (10MB, in sub_FF0EB67C). For 2, it's most likely allocated with srm_malloc, which I'd rather not touch (buffers must be allocated and freed in the same order).

            Looking into it.

            @nikfreak: Can you call FF21835C smemShowFix (70D 111B, no arguments) on the dm-spy-experiments branch, and upload the log?

            1. nikfreak author

              Yay I can do it. Any special place or order where I should call it or won't it matter? Will do it tomorrow though.

              1. Alex

                I did it like this, in "don't click me":

                    void debug_intercept();
                    debug_intercept();
                    void (*smemShowFix)(void) = 0xFF202DE8;
                    smemShowFix();
                    debug_intercept();
                
                  1. Alex

                    Same log in another format:

                    0x40d3c000 ... 0x40dbbfff : BMPVRAM1                  (0.5 MB)
                    0x40dbc000 ... 0x40e3bfff : BMPVRAM2                  (0.5 MB)
                    0x40e3c000 ... 0x4123bfff : FILE_HEADER               (4.0 MB)
                    0x4123c000 ... 0x4143bfff : CAPTURE_WORK1             (2.0 MB)
                    0x4143c000 ... 0x4153bfff : ADAPTER_TRANSFER          (1.0 MB)
                    0x4153c000 ... 0x415651ff : BANK1_FREE1               (0.2 MB)
                    0x41565200 ... 0x415d51ff : DEVELOP_WORK              (0.4 MB)
                    0x415d5200 ... 0x416cbfff : VSHADING_COMP_WORK        (1.0 MB)
                    0x416cc000 ... 0x416dbfff : FENCING_WORK              (0.1 MB)
                    0x416dc000 ... 0x416fbfff : DARKCUR_COMP_WORK         (0.1 MB)
                    0x416fc000 ... 0x416fffff : DCFNO                     (0.0 MB)
                    0x41700000 ... 0x41743fff : ENGINE_MIRROR             (0.3 MB)
                    0x41744000 ... 0x41b2bfff : TUNEDATA                  (3.9 MB)
                    0x41b2c000 ... 0x41d63fff : TUNEDATA2                 (2.2 MB)
                    0x41d64000 ... 0x41dd9aff : STILL_SCAR                (0.5 MB)
                    0x41dd9b00 ... 0x41de83ff : BANK1_FREE2               (0.1 MB)
                    0x41de8400 ... 0x41df33ff : LVMARGE_P_DEF_DATA_3      (0.0 MB)
                    0x41df3400 ... 0x41dfffff : LVMARGE_P_DEF_DATA_CROP   (0.0 MB)
                    0x41e00000 ... 0x41fbffff : EEKO_ADDRESS              (1.8 MB)
                    Gap: 0x6000 (0.0 MB)
                    0x41fc6000 ... 0x41fc6fff : BANK1_FREE3               (0.0 MB)
                    0x41fc7000 ... 0x41fd6bff : LVMARGE_P_DEF_DATA_1      (0.1 MB)
                    0x41fd6c00 ... 0x41fe67ff : LVMARGE_P_DEF_DATA_2      (0.1 MB)
                    0x41fe6800 ... 0x41ffffff : LVMARGE_P_DEF_DATA_ZOOM   (0.1 MB)
                    0x42000000 ... 0x4affffff : MEMORY_MGR1               (144.0 MB)
                    0x4b000000 ... 0x4fffffff : SS_DEVELOP1               (80.0 MB)
                    0x50000000 ... 0x54ffffff : SS_DEVELOP2               (80.0 MB)
                    0x55000000 ... 0x55dfffff : IVA_NVY                   (14.0 MB)
                    0x55e00000 ... 0x5c9fffff : MEMORY_MGR2               (108.0 MB)
                    0x5ca00000 ... 0x5ceab0bf : EXMEM3_AREA1              (4.7 MB)
                    0x5ceab0c0 ... 0x5d4ff0bf : IVA_NVUV                  (6.3 MB)
                    0x5d4ff0c0 ... 0x5dcab0bf : IVA_REF                   (7.7 MB)
                    0x5dcab0c0 ... 0x5ec0ffff : REC_STREAM                (15.4 MB)
                    Gap: -0xe00000 (-14.0 MB)
                    0x5de10000 ... 0x5e80ffff : IMGPLAY_WORK              (10.0 MB)
                    0x5e810000 ... 0x5ec0ffff : IMGPLAY_WORK2             (4.0 MB)
                    Gap: 0x200000 (2.0 MB)
                    0x5ee10000 ... 0x5f21ffff : IMG_VRAM1                 (4.1 MB)
                    0x5f220000 ... 0x5f62ffff : IMG_VRAM2                 (4.1 MB)
                    0x5f630000 ... 0x5fa3ffff : IMG_VRAM3                 (4.1 MB)
                    0x5fa40000 ... 0x5fafffff : IVA_BUFF_ADDRESS          (0.8 MB)
                    0x5fb00000 ... 0x5fc7ffff : WIRELESS_WORK1            (1.5 MB)
                    0x5fc80000 ... 0x5fffffff : WIRELESS_WORK2            (3.5 MB)
                    

                    So, it looks like Canon code assumes the IMGPLAY stuff is not going to be used during capture, so it allocates a SRM buffer at 0x5ca00000. This is going to overlap both IMGPLAY_WORK and IMGPLAY_WORK2.

                    Interesting that IMGPLAY_WORK2 doesn't seem to bother us.

                    I'm thinking to move IMGPLAY_WORK at 0x55000000, over IVA_NVY (which seems to be used by the H.264 encoder). I'll try this change on 5D3 to check whether it's safe to do so.

                    1. Alex

                      On 5D3, IVA_NVY is also reused for SRM buffers.

                      Can you put this in debug_loop_task?

                              int sig = compute_signature(0x55000000, 0xe00000/4);
                              bmp_printf(FONT_MED, 50, 50, "%x ", sig);
                      

                      Then, use (and abuse) the camera in any mode you may think of, take pictures, maybe bursts, record videos and so on. Try to find out which modes use this memory area.

                      1. nikfreak author

                        It's printing always the same onto the LCD:

                        tried with raw_rec, mlv_rec, frsp, LV / movie mode, play mode etc...

                          1. nikfreak author

                            only changes when H264 recording and keeps the last printed value if not recording H264

                        1. nikfreak author

                          and db1886d if raw_rec /mlv_rec is turned off. in that case while recording H264 it changesto different values! Soonly changes when recording H264

  3. nikfreak author

    @a1ex can you have a quick look at these ISO adresses (CMOS[0] / [3]) in movie mode. Can't get CMOS[0] to work in movie mode but works in photo mode. Now got the idea to use CMOS[3] like EOS 6D but that one changes it's address but not it's value. The value always is c00 in movie and photo mode.

    FRAME_CMOS_ISO_SIZE (distance) is the same no matter if I use CMOS[0] or [3] but I could need some assistance on this experiment if you think it could be worth / possible:

    http://pastebin.com/gVxaa4ZY

    I now would want to try CMOS[3] for photo and movie mode but what to put in here based upon pastebin link above?

    CMOS_ISO_BITS = ;
    CMOS_FLAG_BITS = ;
    CMOS_EXPECTED_FLAG = ;
    

    only getting isloess ph/lv(err) whatever I tried to put in... Bear with me. i don't understand that flag stuff.

              1. Alex

                bits=3, flag bits 2, expected flag 3.

                What isoless error are you getting? (the number)

                The configuration looks correct. Maybe you can print the "backup" array to figure it out from there?

                1. nikfreak author

                  No isoless error when using CMOS[0] just when trying to use CMOS[3].Got the idea to use that one but definitely CMOS[0] is the one to use which works in photo mode but produces that "disco effect" in video mode like seen in the video linked above from DeafEyeJedi. I am stuck with that and see no chance in getting it fixed since almost a year

                  1. nikfreak author

                    Just some brainstorming:

                    I got problems with fps override and dualiso movie mode. What things in common do both features share? Could it be something related to one of these?

                    #define VIDEO_PARAMETERS_SRC_3 MEM(0x7CFD0)                     // 0x7cfc4+0x4 nikfreak ok found by alex for old_ICU
                    #define FRAME_ISO (*(uint8_t*)(VIDEO_PARAMETERS_SRC_3+0))
                    #define FRAME_APERTURE (*(uint8_t*)(VIDEO_PARAMETERS_SRC_3+1))
                    #define FRAME_SHUTTER (*(uint8_t*)(VIDEO_PARAMETERS_SRC_3+2))
                    #define FRAME_SHUTTER_TIMER (*(uint16_t*)(VIDEO_PARAMETERS_SRC_3+6))
                    #define FRAME_BV ((int)FRAME_SHUTTER + (int)FRAME_APERTURE - (int)FRAME_ISO)
                    
                    1. nikfreak author

                      One last thing in movie mode:

                      ISO 100 CMOS[0] value 0x3

                      ISO 200 CMOS[0} value 0x27

                      Dualiso 100/200 results in CMOS[0] value 0x23. Frame_CMOS_ISO_SIZE is 46 bytes. Is this theoretically ok?

                    2. Alex

                      Those are used for exposure override, HDR video, gradual exposure... stuff like that. Dual ISO doesn't touch them.

                      For the register values, write the value in binary:

                      0x03 = 0b_000_000_11
                      0x27 = 0b_001_001_11
                      0x23 = 0b_001_000_11
                      

                      See the pattern?

                      Those bytes mean how much the ISO structs (or whatever they are) are apart from each other in memory.

                      1. nikfreak author

                        won't give up on it ofc but at its actual state this PR doesn't enable it anyways (also not fps override). I still vote for a merge of this PR as I don't expect to see a firmware update. Canon simply took 60D's firmware and its version 1.1.1 and placed actual hardware in it so we experiment with it. Proof for that is printed in prop-str.log of 70D(search for 60D in it).

                        If a fw pops up (didn't since release 2013) I will do the job for that one. Again I doubt it due to 80D been out now. Most important would be for me to not being forced to maintain this PR on my own in terms of trying also out new PRs from you. Remember the 100/SL1 is based also upon this PR and this is making it even more difficult for me. I would like to create a PR for the SL1 too but am stuck with that because of this PR.

                        Talking 'bout SL1 that one was easy to do and porting caused no headaches on that one. But that camera even makes it to three firmware revisions (A/B/C) and has a single bug left before I can consider it ready for a merge. I would like to take this opportunity to avoid creating an issue for it and getting offtopic as I already tried to clean caches at some places but that didn't do the trick. The Problem is with the VRAM being "dirty" and I assume I need to call "vram_params_update_if_dirty" in one or more places to get it fixed. I already tried nanomad's ringbuffer but that didn't help, too. See attached two pictures. Half height of the ML menu will look "dirty". . No signs of any dirtyness if ML menu is closed. No signs also on stills/recordings and I didn't notice any negative effect on features but it feels and looks (edmac) wrong.

                        The problem will apear sometimes (not always):

                        • if you switch from photo to LV/movie mode
                        • if you take a picture in LV
                        • if you open up Canon menu and close it in LV/movie mode
                        • if (and ony then) the problem appears the edmac channels also get crazy like seen in the picture below

                        The fix is to switch to playback mode and back.

                        Half Height dirtyness screenshot

                        EDMACs going crazy

                        Any idea what to try in terms of VRAM?

                        1. Alex

                          I'll take a look at 70D after merging some important fixes like lua_fix and re-enabling the 600D port. We can investigate FPS and dual ISO and whatever else is not working after merging.

                          For the SL1, I have an idea. After finishing drawing the menu, call clean_d_cache(). There is an example of a somewhat similar problem in zebra.c, draw_overlays_playback, visible on most cameras in particular if you enable cropmarks and show them in playback mode.

                          1. nikfreak author

                            SL1: clean_d_cache didn't help (tried everywhere I could like: piggyback, open/close menu, redraw etc..). Started from scratch looking for stubs / consts and disabled lots of features but nope. Tried harder even by increasing / decreasing AllocateMemory and so on.

                            Finally after activating gui_events in debug menu with 300ms delay I noticed the menu will open up cleanly and become dirty afterwards. Then took a closer look after stumbling upon an old thread from you. Investigating the menu code I tried it w/o DOUBLE_BUFFERING and that solves the problem but flickers. ATM I think it's bmp_flip or something related to bmp_draw_to_idle, bmp_vram*...

                            1. nikfreak author

                              100D / SL1 problem solved btw. Awaiting this PR to get merged so I can create a new one for 100D ;D