Ring returned in riak_core_ring:all_owners(Ring) contains references to removed nodes

[asyncmind0]

Partitions = [P || {P, 'dev1@127.0.0.1'} <- riak_core_ring:all_owners(Ring)].

following instructions for repair in here https://www.tiot.jp/riak-docs/riak/kv/2.2.6/using/repair-recovery/repairs/#repairing-partitions

I found that in my cluster there is a lot of references to old nodes
My cluster only has the following valid nodes

dev-riak1.example.com
dev-riak2.example.com
dev-riak3.example.com
dev-riak4.example.com
dev-riak5.example.com

But the Ring contains

[{'riak@dev-riak5.example.com',{793,63760267758}},
             {'riak@dev-riak0.example.com',{2657,63759791951}},
             {'riak@dev-riak1.example.com',{2492,63759791951}},
             {'riak@dev-riak2.example.com',{2257,63759785044}},
             {'riak@dev-riak3.example.com',{1652,63759445321}},
             {'riak@dev-riak4.example.com',{2001,63760226110}},
             {'riak@devold0.example.com',{4652,63673217155}},
             {'riak@devold1.example.com',{7409,63685197199}},
             {'riak@devold10.example.com',{3152,63721849744}},
             {'riak@devold11.example.com',{711,63683938533}},
             {'riak@devold12.example.com',{1503,63721056942}},
             {'riak@devold2.example.com',{4712,63721084448}},
             {'riak@devold3.example.com',{5190,63721701233}},
             {'riak@devold4.example.com',{1700,63660968738}},
             {'riak@devold5.example.com',{6339,63673205425}},
             {'riak@devold6.example.com',{2139,63673186576}},
             {'riak@devold7.example.com',{2108,63676347325}},
             {'riak@devold8.example.com',{1482,63721608275}},
             {'riak@devold9.example.com',{745,63683941378}}],
            {64,
             [{0,'riak@dev-riak5.example.com'},
              {22835963083295358096932575511191922182123945984,
               'riak@dev-riak1.example.com'},
              {45671926166590716193865151022383844364247891968,
               'riak@dev-riak2.example.com'},
              {68507889249886074290797726533575766546371837952,
               'riak@dev-riak3.example.com'},
              {91343852333181432387730302044767688728495783936,
               'riak@dev-riak4.example.com'},
              {114179815416476790484662877555959610910619729920,
               'riak@dev-riak5.example.com'},
              {137015778499772148581595453067151533092743675904,
               'riak@dev-riak1.example.com'},
              {159851741583067506678528028578343455274867621888,
               'riak@dev-riak2.example.com'},
              {182687704666362864775460604089535377456991567872,
               'riak@dev-riak3.example.com'},
              {205523667749658222872393179600727299639115513856,
               'riak@dev-riak4.example.com'},
              {228359630832953580969325755111919221821239459840,
               'riak@dev-riak5.example.com'},
              {251195593916248939066258330623111144003363405824,
               'riak@dev-riak1.example.com'},
              {274031556999544297163190906134303066185487351808,
               'riak@dev-riak2.example.com'},
              {296867520082839655260123481645494988367611297792,
               'riak@dev-riak3.example.com'},
              {319703483166135013357056057156686910549735243776,
               'riak@dev-riak4.example.com'},
              {342539446249430371453988632667878832731859189760,
               'riak@dev-riak5.example.com'},
              {365375409332725729550921208179070754913983135744,
               'riak@dev-riak1.example.com'},
              {388211372416021087647853783690262677096107081728,
               'riak@dev-riak2.example.com'},
              {411047335499316445744786359201454599278231027712,
               'riak@dev-riak3.example.com'},
              {433883298582611803841718934712646521460354973696,
               'riak@dev-riak4.example.com'},
              {456719261665907161938651510223838443642478919680,
               'riak@dev-riak5.example.com'},
              {479555224749202520035584085735030365824602865664,...},
              {...}|...]},
            {dict,6,16,16,8,80,48,
                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                  {{[],[],[],[],
                    [[riak_core_ring_tainted|{meta_entry,true,63760267758}]],
                    [[symbolic_clustername|
                      {meta_entry,"dev-riak",63739630764}]],
                    [],
                    [[riak_repl_ring|{meta_entry,{dict,...},63759488652}]],
                    [[{bucket,<<"_yz_"...>>}|{meta_entry,[...],...}]],
                    [],[],
                    [[{...}|...]],
                    [],[],...}}},
            {'riak@devold5.example.com',{1479,946122,353764}},
            [],
            [{'riak@dev-riak0.example.com',{invalid,[{'riak@dev-riak0.example.com',{2209,
                                                                                              63759791951}},
                                                          {'riak@dev-riak2.example.com',{1,63759785044}},
                                                          {'riak@devold10.example.com',{1,63720652846}}],
                                                         []}},
             {'riak@dev-riak1.example.com',{valid,[{'riak@dev-riak1.example.com',{2206,
                                                                                            63759444688}},
                                                        {'riak@devold10.example.com',{1,63720692674}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true,false]},
                                                                               {{riak_core,staged_joins},[true,false]},
                                                                               {{riak_core,vnode_routing},[proxy|...]},
                                                                               {{riak_kv,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'riak@dev-riak2.example.com',{valid,[{'riak@dev-riak2.example.com',{2069,
                                                                                            63759444717}},
                                                        {'riak@devold10.example.com',{1,63720709605}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true,false]},
                                                                               {{riak_core,staged_joins},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'riak@dev-riak3.example.com',{valid,[{'riak@dev-riak3.example.com',{1422,
                                                                                            63759445321}},
                                                        {'riak@devold10.example.com',{1,63721340231}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true,false]},
                                                                               {{riak_core,security},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'riak@dev-riak4.example.com',{valid,[{'riak@dev-riak4.example.com',{1802,
                                                                                            63760226110}},
                                                        {'riak@devold10.example.com',{1,63721340231}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true,false]},
                                                                               {{riak_core,resizable_ring},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'riak@dev-riak5.example.com',{valid,[{'riak@dev-riak5.example.com',{681,
                                                                                            63760267758}},
                                                        {'riak@dev-riak1.example.com',{1,63759791951}},
                                                        {'riak@dev-riak0.example.com',{3,63754251063}}],
                                                       [{'$riak_capabilities',[{{riak_control,member_info_version},
                                                                                [v1,v0]},
                                                                               {{riak_core,bucket_types},[true,false]},
                                                                               {{riak_core,fold_req_version},[v2,v1]},
                                                                               {{riak_core,net_ticktime},[true|...]},
                                                                               {{riak_core,...},[...]},
                                                                               {{...},...},
                                                                               {...}|...]},
                                                        {gossip_vsn,2},
                                                        {participate_in_coverage,true}]}},
             {'riak@devold0.example.com',{invalid,[{'riak@devold0.example.com',{3854,
                                                                                  63673217155}},
                                                    {'riak@devold1.example.com',{5,63673217156}},
                                                    {'riak@devold5.example.com',{1,63660953533}}],
                                                   []}},
             {'riak@devold1.example.com',{invalid,[{'riak@devold0.example.com',{1,
                                                                                  63660961602}},
                                                    {'riak@devold1.example.com',{6209,63685197199}},
                                                    {'riak@devold10.example.com',{6,63685197207}},
                                                    {'riak@devold2.example.com',{1,63683442945}},
                                                    {'riak@devold5.example.com',{6,63660932914}}],
                                                   []}},
             {'riak@devold10.example.com',{invalid,[{'riak@dev-riak0.example.com',{2,
                                                                                         63721849745}},
                                                     {'riak@devold1.example.com',{4,63683893655}},
                                                     {'riak@devold10.example.com',{2059,63721849744}}],
                                                    []}},
             {'riak@devold11.example.com',{invalid,[{'riak@devold1.example.com',{3,
                                                                                   63683938543}},
                                                     {'riak@devold11.example.com',{525,63683938533}}],
                                                    []}},
             {'riak@devold12.example.com',{invalid,[{'riak@devold1.example.com',{1,
                                                                                   63673234572}},
                                                     {'riak@devold10.example.com',{3,63721056943}},
                                                     {'riak@devold12.example.com',{1263,...}}],
                                                    []}},
             {'riak@devold2.example.com',{invalid,[{'riak@devold0.example.com',{2,
                                                                                  63660967356}},
                                                    {'riak@devold1.example.com',{2,...}},
                                                    {'riak@devold10.example.com',{...}},
                                                    {'riak@devold2.example.com',...},
                                                    {...}],
                                                   []}},
             {'riak@devold3.example.com',{invalid,[{'riak@devold0.example.com',{3,...}},
                                                    {'riak@devold10.example.com',{...}},
                                                    {'riak@devold3.example.com',...},
                                                    {...}],
                                                   []}},
             {'riak@devold4.example.com',{invalid,[{'riak@devold0.example.com',{...}},
                                                    {'riak@devold4.example.com',...},
                                                    {...}],
                                                   []}},
             {'riak@devold5.example.com',{invalid,[{'riak@devold0.example.com',...},
                                                    {...}|...],
                                                   []}},
             {'riak@devold6.example.com',{invalid,[{...}|...],[]}},
             {'riak@devold7.example.com',{invalid,[...],...}},
             {'riak@devold8.example.com',{invalid,...}},
             {'riak@devold9.example.com',{...}}],
            'riak@dev-riak1.example.com',
            [{'riak@dev-riak0.example.com',[{'riak@dev-riak0.example.com',{2656,
                                                                                     63759585618}},
                                                 {'riak@dev-riak1.example.com',{2477,63759444688}},
                                                 {'riak@dev-riak2.example.com',{2256,63759444717}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{1960,63759445973}},
                                                 {'riak@dev-riak5.example.com',{656,63754251063}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,63685197199}},
                                                 {'riak@devold10.example.com',{3152,63721849744}},
                                                 {'riak@devold11.example.com',{711,63683938533}},
                                                 {'riak@devold12.example.com',{1503,63721056942}},
                                                 {'riak@devold2.example.com',{4712,63721084448}},
                                                 {'riak@devold3.example.com',{5190,...}},
                                                 {'riak@devold4.example.com',{...}},
                                                 {'riak@devold5.example.com',...},
                                                 {...}|...]},
             {'riak@dev-riak1.example.com',[{'riak@dev-riak0.example.com',{2657,
                                                                                     63759791951}},
                                                 {'riak@dev-riak1.example.com',{2492,63759791951}},
                                                 {'riak@dev-riak2.example.com',{2257,63759785044}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{2001,63760226110}},
                                                 {'riak@dev-riak5.example.com',{789,63760267758}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,63685197199}},
                                                 {'riak@devold10.example.com',{3152,63721849744}},
                                                 {'riak@devold11.example.com',{711,63683938533}},
                                                 {'riak@devold12.example.com',{1503,63721056942}},
                                                 {'riak@devold2.example.com',{4712,...}},
                                                 {'riak@devold3.example.com',{...}},
                                                 {'riak@devold4.example.com',...},
                                                 {...}|...]},
             {'riak@dev-riak2.example.com',[{'riak@dev-riak0.example.com',{2657,
                                                                                     63759791951}},
                                                 {'riak@dev-riak1.example.com',{2492,63759791951}},
                                                 {'riak@dev-riak2.example.com',{2257,63759785044}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{2001,63760226110}},
                                                 {'riak@dev-riak5.example.com',{789,63760267758}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,63685197199}},
                                                 {'riak@devold10.example.com',{3152,63721849744}},
                                                 {'riak@devold11.example.com',{711,63683938533}},
                                                 {'riak@devold12.example.com',{1503,...}},
                                                 {'riak@devold2.example.com',{...}},
                                                 {'riak@devold3.example.com',...},
                                                 {...}|...]},
             {'riak@dev-riak3.example.com',[{'riak@dev-riak0.example.com',{2657,
                                                                                     63759791951}},
                                                 {'riak@dev-riak1.example.com',{2492,63759791951}},
                                                 {'riak@dev-riak2.example.com',{2257,63759785044}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{2001,63760226110}},
                                                 {'riak@dev-riak5.example.com',{789,63760267758}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,63685197199}},
                                                 {'riak@devold10.example.com',{3152,63721849744}},
                                                 {'riak@devold11.example.com',{711,...}},
                                                 {'riak@devold12.example.com',{...}},
                                                 {'riak@devold2.example.com',...},
                                                 {...}|...]},
             {'riak@dev-riak4.example.com',[{'riak@dev-riak0.example.com',{2657,
                                                                                     63759791951}},
                                                 {'riak@dev-riak1.example.com',{2492,63759791951}},
                                                 {'riak@dev-riak2.example.com',{2257,63759785044}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{2001,63760226110}},
                                                 {'riak@dev-riak5.example.com',{789,63760267758}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,63685197199}},
                                                 {'riak@devold10.example.com',{3152,...}},
                                                 {'riak@devold11.example.com',{...}},
                                                 {'riak@devold12.example.com',...},
                                                 {...}|...]},
             {'riak@dev-riak5.example.com',[{'riak@dev-riak0.example.com',{2657,
                                                                                     63759791951}},
                                                 {'riak@dev-riak1.example.com',{2492,63759791951}},
                                                 {'riak@dev-riak2.example.com',{2257,63759785044}},
                                                 {'riak@dev-riak3.example.com',{1652,63759445321}},
                                                 {'riak@dev-riak4.example.com',{2001,63760226110}},
                                                 {'riak@dev-riak5.example.com',{789,63760267758}},
                                                 {'riak@devold0.example.com',{4652,63673217155}},
                                                 {'riak@devold1.example.com',{7409,...}},
                                                 {'riak@devold10.example.com',{...}},
                                                 {'riak@devold11.example.com',...},
                                                 {...}|...]},
             {'riak@devold0.example.com',[{'riak@devold0.example.com',{4652,
                                                                         63673217155}},
                                           {'riak@devold1.example.com',{5501,63673217065}},
                                           {'riak@devold10.example.com',{56,63673217065}},
                                           {'riak@devold2.example.com',{3325,63673217065}},
                                           {'riak@devold3.example.com',{3763,63673217065}},
                                           {'riak@devold4.example.com',{1700,63660968738}},
                                           {'riak@devold5.example.com',{6339,...}},
                                           {'riak@devold6.example.com',{...}},
                                           {'riak@devold7.example.com',...},
                                           {...}|...]},
             {'riak@devold1.example.com',[{'riak@devold0.example.com',{4652,
                                                                         63673217155}},
                                           {'riak@devold1.example.com',{7409,63685197199}},
                                           {'riak@devold10.example.com',{1742,63685177941}},
                                           {'riak@devold11.example.com',{711,63683938533}},
                                           {'riak@devold12.example.com',{940,63685175606}},
                                           {'riak@devold2.example.com',{4137,...}},
                                           {'riak@devold3.example.com',{...}},
                                           {'riak@devold4.example.com',...},
                                           {...}|...]},
             {'riak@devold10.example.com',[{'riak@dev-riak0.example.com',{1072,
                                                                                63721811710}},
                                            {'riak@dev-riak1.example.com',{585,63721849665}},
                                            {'riak@dev-riak2.example.com',{498,63721806643}},
                                            {'riak@dev-riak3.example.com',{329,63721823494}},
                                            {'riak@dev-riak4.example.com',{422,...}},
                                            {'riak@devold0.example.com',{...}},
                                            {'riak@devold1.example.com',...},
                                            {...}|...]},
             {'riak@devold11.example.com',[{'riak@devold0.example.com',{4652,
                                                                          63673217155}},
                                            {'riak@devold1.example.com',{6756,63683936507}},
                                            {'riak@devold10.example.com',{1259,63683894630}},
                                            {'riak@devold11.example.com',{711,...}},
                                            {'riak@devold12.example.com',{...}},
                                            {'riak@devold2.example.com',...},
                                            {...}|...]},
             {'riak@devold12.example.com',[{'riak@dev-riak0.example.com',{784,
                                                                                63720908410}},
                                            {'riak@dev-riak1.example.com',{187,63720952694}},
                                            {'riak@dev-riak2.example.com',{103,...}},
                                            {'riak@devold0.example.com',{...}},
                                            {'riak@devold1.example.com',...},
                                            {...}|...]},
             {'riak@devold2.example.com',[{'riak@dev-riak0.example.com',{784,
                                                                               63720908410}},
                                           {'riak@dev-riak1.example.com',{187,...}},
                                           {'riak@dev-riak2.example.com',{...}},
                                           {'riak@devold0.example.com',...},
                                           {...}|...]},
             {'riak@devold3.example.com',[{'riak@dev-riak0.example.com',{910,...}},
                                           {'riak@dev-riak1.example.com',{...}},
                                           {'riak@dev-riak2.example.com',...},
                                           {...}|...]},
             {'riak@devold4.example.com',[{'riak@devold0.example.com',{...}},
                                           {'riak@devold1.example.com',...},
                                           {...}|...]},
             {'riak@devold5.example.com',[{'riak@devold0.example.com',...},
                                           {...}|...]},
             {'riak@devold6.example.com',[{...}|...]},
             {'riak@devold7.example.com',[...]},
             {'riak@devold8.example.com',...},
             {...}],
            [{'riak@dev-riak0.example.com',{43,63759785047}},
             {'riak@devold10.example.com',{528,63721849745}},
             {'riak@devold1.example.com',{336,63685197207}},
             {'riak@devold0.example.com',{259,63665215960}},
             {'riak@devold5.example.com',{120,63660958341}}]}

[martinsumner]

Apologies that this wan't answered before.

As far as I can see all the extra nodes are marked as invalid and so will play no actual role in the ring. I've just double-checked in a test cluster, and it looks like nodes that have left will remain referenced in the ring as invalid.

The precise reasons why this tombstone state is never garbage collected, I don't know. I can't find this documented anywhere in the code, or in the riak_core wiki.

I don't believe there currently exists a function to clean out this history. It may be relatively simple to extend riak_core to allow this to happen, but unless the history is causing an active problem, I'm not sure this would ever be prioritised.