Add counters to foreach loops

[Aaron-Junker]

The problem

I often find myself writing something of the form like this:

int i = 0;
foreach (var item in a)
{
    Console.WriteLine("Item " + i + ":" + item);
    i++;
}

When using arrays you can just do:

for (int j = 0; j < a.Length; j++)
{
    Console.WriteLine("Item " + j + ":" + a[j]);
}

But this does not work for any kind of enumerators as...

1. a needs to have a Length property
2. And a needs to have an accessor

Proposal

Add the possibility to define an optional counter in the foreach loop definition:

foreach (int i = 0; var item in a)
{
    Console.WriteLine("Item " + i + ":" + item);
}

(Maybe the =0 part could be omitted, but I don't know if it fits the language.)

The counter variable is initialized at the beginning of the foreach loop and incremented by 1 after every iteration.

The type of the counter has to be IBinaryInteger<TSelf>

Syntax

 foreach_statement
-    : 'foreach' '(' ref_kind? local_variable_type identifier 'in' 
+    : 'foreach' '(' foreach_counter? ref_kind? local_variable_type identifier 'in' 
       expression ')' embedded_statement
     ;

+foreach_counter
+    : local_variable_declaration ';'
+    ;

[colejohnson66]

Theoretically, the new Enumerable.Index could be used here, but that has a performance hit and can’t be used with spans.

I personally like this proposal as it could be trivially lowered to a either a dedicated counter or a for loop:

for (int i = 0; i < src.Length; i++)
{
    T element = src[i];
    // …
}

[GrabYourPitchforks]

What is the expected behavior if the number of elements overflows the max value of the counter variable?

[colejohnson66]

I’d assume the same as if you kept track of the counter yourself: it overflows and wraps around. IIRC, that’s what Enumerable.Index will do in .NET 9.

[GrabYourPitchforks]

FYI Enumerable.Index will throw an OverflowException and terminate the iterator. I don't mean to imply that some behavior is right or wrong, just that the expectation should be documented.

[robertmclaws]

I feel like the whole reason ForEach exists is because sometimes you just want to iterate without a counter. If you need a counter, For loops already exists

Making a ForEach look almost exactly like a For would seem to add complexity without payoff, as For loops will always perform better if you need a counter.

[dersia]

Also if you have an enumerator that is either async (IAsyncEnumerator) or just lazyly iterates and you first have to call Count() on it to use it in a for loop, it would iterate the whole thing already just to give you a length so you have to iterate over it again within the for loop. and there are scenarios where you can't iterate over it twice i.e. data is coming from a (IO)-Stream.

perfwise itering twice is also not a good solution.

[CyrusNajmabadi]

@dersia you don't need to do this. you can just do:

async IAsyncEnumerable<(T value, int index)> WithIndex<T>(this IAsyncEnumerable<T> values)
{
    var index = 0;
    await foreach (var value in values)
        yield return (value, index++);
}

now you can do:

await foreach (var (value, index) in values.WithIndex())
    ...

[dersia]

@CyrusNajmabadi

thanks, that's basically what I do. I was just trying to answer to @robertmclaws comment about why there would still be a need, even if there are for loops. it wasn't much about in favor of this proposal, more about giving reasons, when not to use for loops 😊

[colejohnson66]

@CyrusNajmabadi Custom IEnumerable implementations, such as dotnet 9's new Enumerable.Index extension, have overhead. The compiler has special lowerings for things like spans and arrays that ensure fast operation. But, once you throw them into a dedicated IEnumerable, you lose that optimization. Unless you're suggesting the runtime be altered to pattern recognize these things, but I doubt that.

[HaloFour]

If you're enumerating over an async enumerable the overhead of virtual calls is very likely background noise. However, for normal IEnumerable<T>, you could write the extension method to return a struct that adheres to the enumerable/enumerator pattern, which should be very close to as performant as managing an index variable manually, which always remains an option.

[HaloFour]

See: #1584

[zms9110750]

IEnumerable<int> enumer = [4, 8, 10, 14];

foreach (var (item, index) in enumer.Select((item, index) => (item, index)))
{
	Console.WriteLine("Item " + index + ":" + item);
}

[xin9le]

The addition of the Enumerable.Index() extension method from .NET 9 is expected to significantly alleviate this issue.

foreach (var (index, item) in source.Index())
{ }

• https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.index

[siegfriedpammer]

I was curious if this is really worth it and have created benchmark, which tries manual increment vs. .NET 9's Index (internally an iterator state-machine) vs. a custom struct-based implementation.

Here's the results on my machine:

Method	N	Mean	Error	StdDev	Ratio
ManualIncrement	1000	2.657 us	0.0504 us	0.0472 us	1.00
IndexMethodNet9	1000	9.946 us	0.0085 us	0.0076 us	1.00
IndexMethodStruct	1000	3.796 us	0.0759 us	0.1159 us	1.00
ManualIncrement	10000	26.218 us	0.0776 us	0.0688 us	1.00
IndexMethodNet9	10000	98.476 us	0.1668 us	0.1560 us	1.00
IndexMethodStruct	10000	37.779 us	0.1026 us	0.0857 us	1.00

Of course the benchmark is rather simple and there is still room for improvement in my custom implementation, which I would like to defer to someone more knowledgeable in IEnumerator optimization tricks.

My personal conclusion is that a language feature is definitely NOT worth the effort, because the difference is minimal and existing features can be used paired with JIT magic to achieve very good results.

I hope that LDM agrees with this and we can get some optimizations done on the Enumerable.Index implementation, which is currently not optimized compared to other parts of LINQ, which perform a lot of magic tricks.

Here the code I used for my benchmark. Feel free to experiment!

using System.Collections;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;

[SimpleJob(RuntimeMoniker.Net80, baseline: true)]
public class IndexEnumeration
{
    IEnumerable<object> source;
    object[] target;

    [Params(1000, 10000)]
    public int N;

    [GlobalSetup]
    public void Setup()
    {
        source = Enumerable.Range(1, N).Cast<object>().ToArray();
        target = new object[N];
    }

    [Benchmark]
    public void ManualIncrement()
    {
        int i = 0;
        foreach (var item in source)
        {
            target[i++] = item;
        }
    }

    [Benchmark]
    public void IndexMethodNet9()
    {
        foreach (var (i, item) in source.Index())
        {
            target[i] = item;
        }
    }

    [Benchmark]
    public void IndexMethodStruct()
    {
        foreach (var (i, item) in source.Index2())
        {
            target[i] = item;
        }
    }
}

public static partial class EnumerableExt
{
    public static IEnumerable<(int Index, TSource Item)> Index<TSource>(this IEnumerable<TSource> source)
    {
        if (source == null)
        {
            ArgumentNullException.ThrowIfNull(nameof(source));
        }

        if (source is ICollection<TSource> { Count: 0 })
        {
            return Array.Empty<(int Index, TSource Item)>();
        }

        return IndexIterator(source);
    }

    private static IEnumerable<(int Index, TSource Item)> IndexIterator<TSource>(IEnumerable<TSource> source)
    {
        int index = -1;
        foreach (TSource element in source)
        {
            checked
            {
                index++;
            }

            yield return (index, element);
        }
    }

    public static IEnumerable<(int Index, TSource Item)> Index2<TSource>(this IEnumerable<TSource> source)
    {
        if (source == null)
        {
            ArgumentNullException.ThrowIfNull(nameof(source));
        }

        if (source is ICollection<TSource> { Count: 0 })
        {
            return Array.Empty<(int Index, TSource Item)>();
        }

        return new IndexEnumerator2<TSource>(source);
    }

    struct IndexEnumerator2<TSource> : IEnumerator<(int Index, TSource Item)>, IEnumerable<(int Index, TSource Item)>
    {
        int index;
        IEnumerator<TSource> source;

        public IndexEnumerator2(IEnumerable<TSource> source)
        {
            this.index = -1;
            this.source = source.GetEnumerator();
        }

        (int Index, TSource Item) IEnumerator<(int Index, TSource Item)>.Current => (index, source.Current);

        object IEnumerator.Current => (index, source.Current);

        void IDisposable.Dispose()
        {
            source.Dispose();
        }

        IEnumerator<(int Index, TSource Item)> IEnumerable<(int Index, TSource Item)>.GetEnumerator()
        {
            // TODO implement correctly
            return this;
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            // TODO implement correctly
            return this;
        }

        bool IEnumerator.MoveNext()
        {
            checked { index++; }
            return source.MoveNext();
        }

        void IEnumerator.Reset()
        {
            index = -1;
            source.Reset();
        }
    }
}